Add practical tips for downsampling. #3340
Conversation
|
Can we wait for Marci to submit her update, then port these tips to the new structure? |
| ## Practical tips | ||
|
|
||
| Downsampling requires reading and indexing the contents of a backing index. The following guidelines can help you get the most out of it. | ||
|
|
There was a problem hiding this comment.
Do we need a note about rollover? To avoid creating backing indices that are too big..
There was a problem hiding this comment.
I have been going back and forth for this. For ILM it's easy because it's part of the policy, for data stream lifecycle, I would suggest that if we really think that it should be less maybe we should set it to something less. Right?
There was a problem hiding this comment.
You mean, update the default? We can do that at a later point, but what about older versions, or ILM configurations with existing rollover overrides? It could still help to suggest a best practice here.
There was a problem hiding this comment.
Yes, we could update the default, that would apply on all version unless the user chose to overwrite it. I restructure it a bit so we can have ILM focused recommendations. But if we think it should be reduced, we should consider updating the default for DLM as well.
There was a problem hiding this comment.
Let's file a tracking issue for this, so that we don't forget.
It is created on the updated downsampling page. Right? |
🔍 Preview links for changed docs |
Co-authored-by: Kostas Krikellas <[email protected]>
Co-authored-by: Kostas Krikellas <[email protected]>
|
|
||
| ### Choosing the downsampling interval | ||
|
|
||
| When choosing the downsampling interval, you need to consider the original sampling rate of your measurements. Ideally, you would like an interval that would reduce your number of documents by a significant amount. For example, if a sensor sends data every 10 seconds downsampling to 1 minute would reduce the number of documents by 83%, compared to downsampling to 5 minutes by 96%. |
There was a problem hiding this comment.
| When choosing the downsampling interval, you need to consider the original sampling rate of your measurements. Ideally, you would like an interval that would reduce your number of documents by a significant amount. For example, if a sensor sends data every 10 seconds downsampling to 1 minute would reduce the number of documents by 83%, compared to downsampling to 5 minutes by 96%. | |
| When choosing the downsampling interval, you need to consider the original sampling rate of your measurements. Ideally, you would like an interval that would reduce your number of documents by a significant amount. For example, if a sensor sends data every 10 seconds, downsampling to 1 minute would reduce the number of documents by 83%, compared to downsampling to 5 minutes by 96%. |
Co-authored-by: Marci W <[email protected]>
|
|
||
| ### Reduce the index size (ILM only) | ||
|
|
||
| When configuring an ILM policy with downsampling, use the [rollover action](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-rollover.md) in the `hot` phase to control index size. Using smaller indices helps to minimize the impact of downsampling on a cluster's performance. |
There was a problem hiding this comment.
I think it's important here to say use and not define. When writing an ILM policy a user needs to define a rollover action no matter what. However, if they are using downsampling, they can consider using this to reduce the size of their index. I want us to be careful to not imply that a user needs to define it only if they are trying to reduce the size. @marciw does this make sense?
| When using [index lifecycle management](/manage-data/lifecycle/index-lifecycle-management.md) (ILM), you can define at most one downsampling round in each of the following phases: | ||
|
|
||
| - `hot` phase: Runs after the [index time series end time](elasticsearch://reference/elasticsearch/index-settings/time-series.md#index-time-series-end-time) passes | ||
| - `warm` phase: Runs after the `min_age` time (starting to count after the rollover and respecting the [index time series end time](elasticsearch://reference/elasticsearch/index-settings/time-series.md#index-time-series-end-time)) |
There was a problem hiding this comment.
Does this read well @marciw ? Considering that we want to stress that rollover marks "the start of time"?
|
I am adding @leontyevdv as a reviewer because he recently watched a tutorial on how to configure TSDS and he can tell us how well it reads for a user. |
In this PR, we propose to add practical tips for downsampling. For now this includes, a guideline on how to choose the downsampling interval. And then specifically for ILM, an explanation on how downsampling relates with tiers. After elastic/elasticsearch#135834, we should also add here the option to disable force merge.